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ABSTRACT 

This report specifies the contributions of the Bureau 
of Occupational Education Research (BOER) towards establishing an 
education evaluc tion system in the New York State Education 
Department. Section I shows what BOER has done since 1968 in 
developing an education evaluation system* Thirteen documented 
examples are given to demonstrate samples of the understandings, 
insights, personnel, skills, media, data, objective measures, 
applications, systems, and other resources developed by BOER that can 
be integrated into almost any type of education evaluation system 
that is likely to emerge* Section II goes beyond the obvious 
requirements of more time, money, and manpower to make any such 
system workable by suggesting the need to coordinate existing 
departmental functions through a number of organizational **linking 
pins" and the need to ••tool up«* existing resources to produce a 
departmentwide effort to bridge the evaluation gap in a humanistic 
fashion* Specific examples and directives are given to spell out 
exactly what is meant by "linking pins" and "tooling up*" The report 
concludes that evaluation is best achieved when the goals of a 
particular function or unit are expressed so clearly and succinctly 
that even a stranger walking in off the street can get a clear idea 
of what is going on and how well it is being done* (Author/DEP) 
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SUMMARY 



This report specifies the contributions of the Bureau of Occupa- 
tional Education Research (BOER) towards establishing an education 
evaluation system in the New York State Education Department. In the 
Introduction, the point of view of BOER is shown to be research with 
integrated evaluation in the context of Occupational Education; this 
is made specific by reference to the primary functions of BOER, 

Section I shows what BOER has done since 1968 in developing an 
education evaluation system. Thirteen documented examples are given 
to demonstrate samples of the understandings, insights, personnel, skill 
media, data, objective measures, applications, systems, and other re- 
sources developed by BOER that can be integrated into almost any type of 
education evaluation system that is likely to emerge. 

Section II goes beyond the obvio»*s requirements of more time, money 
and manpower to make any such system workable by suggesting the need to 
coordinate existing departmental functions through a number of organic 
zational "linking pins" and the need to "tool up" existing resources 
CO produce a departmentv7ide effort to bridge the evaluation gap in a 
humanistic fashion. Specific examples and directives are given to spell 
out exactly what is meant by "linking pins" and "tooling up." 

The report concludes that evaluation is best achieved when the 
goals of a particular function or unit are expressed so clearly and 
succinctly that even a stranger walking in off the street can get a 
clear idea of what is going on and how well it is being done. 
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INTRODUCTION 



THE CONTRIBUTIONS OF THE BUREAU OF OCCUPATIONAL EDUCATION RESEARCH 
TOWARDS ESTABLISHING AN EDUCATION EVALUATION SYSTEM 

Objectives 

This report tries to answer two questions: 

1. What has the Bureau of Occupational Education Research (BOER) 
done since 1968 tov/ards establishing an education evaluation 
system? (Section 1) 

2. What still needs to be done? Specifically, how much more 
time, money, manpov,'er, and other resources are needed for 
optimum development:? (Section 11) 

These objectives rest upon two assumptions: 

1. That the BOER is not starting from scratch in its efforts 
to build an education evaluation system. 

2. That components of this proposed system exist in BOER and 
in other units of the State Education Department (SED). 

Seen in this perspective, the establishment of an Education 
Evaluation System does not have to concern itself so much with the 
creation of a system from zero as much as with the coordination of 
a number of activities and programs already in existence in a non- 
interacting framework. 



Point of Viov; of BOER 

The Bureau of Occupational Education Research (BOER) is the Research 
Coordinating Unit (RCU) for New York State. Its stress on research and 
evaluation arises since the unit is within the State Education Department 
structure undar che administrative direction of the Associate Commissioner 
for Reser,rch and Evaluation. Its stress on occupational education arises 
since its assignments, responsibilities, and functions are specified by 
the Assistant Commissioner for Occupational Education, 

Functions 

This general description can be made more specific by consideration 
of some of the primary functions of BOER and R.CU: 

1. Administer and manage occupational education research 
and projects; administer and manage developmental, pilot, 
and innovative programs funded under the Vocational 
Education Act by grant or contract to nonprofit organizations, 

2. Initiate "in-house" research and evaluation activity, 

3. Provide consultant services within and without the Department, 

4. Disseminate information and research findings in occupational 
education, 

5. Coordinate occupational education research and evaluation with 
other research activities within and outside the State, 



Section I — What BOER has done to establish an education evaluation 
system. 

Part A — Increase understanding of evaluation as a management tool. 



Example 1 — Manapement by Objectives ; The Massachusetts-New York- 
Pennsylvenia Evaluation Service Center for Occupational 
Education (ESCOE) fulfills three functions : 

(Within Local Educational Agencies (LEAs)) 

a# Statement of Occupational Curricula in 
Behavioral Objective Terminology, 

b, In-Service Training of Occupational Education 
Staff in Curriculum Writing and Revision. 

Cm Development of Test Items Which are Valid 
Measures of Performance. 

Example 2 — Dissemination j^^st:em ; As part of the ERIC System and 
the PREP System, BOER fulfills two functions: 

(Within LEAs and SED) 

a. Dissemination of the latest results of available 
research in occupational education. 

b. Periodic indepth treatment of topics of interest 
to occupational educators. 

Example 3 — Conference Evaluation : As part of its consultant 

services, BOER helps various units in the State Edu- 
cation Department in these areas: 



(Within SED) 



a. Planning and development of built-in criteria 
which insure that evaluation is not something 
done after the conference is over. 

b. Demonstration of how properly conceived conference 
goals implement the attainment of objectives that 
are clearly spelled out and measurable. 

c. Specification of certain general criteria which 
help answer two questions: "What are we trying to 
do?" and "How will we know when we have succeeded? 
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Part B — Secure personnel to carry out evaluation tasks. 

Example 4 - Highly Skilled Evaluation Experts 
(Coordination between SED and LEAs) 

Two Research and Development Centers for Occupational 
Research have been established at City University of 
New York and at Cornel . University under the direction 
of BOER Lo fulfill certain key functions: 

a. Adaptation of research coordinated by RCU to urban 
(CUNY) and upstate (Cornell) occupational educational 
needs. 

b. Training of graduate and undergraduate students in 
those areas which constitute occupational research. 

c. Recruitment of specialists on a consultant basis 
when such assistance is necessary for particular 
problem areas, 

d. Training and updating of SED, LEA, and administrative 
personnel when new developments break in occupational 
education. 

Example 5 — Highly MoLivated Classroom Teachers and Administrators 
(Within LEAs) 

Mini-grants (usually about $3,000 each) have been used 
to stimulate certain kinds of activities which constitute 
the heart of the daily grind of evaluation tasks: 

a. Ability to specify and pinpoint a specific area 
under the influence of the investigators and the 
ability to mark off measurable portions thereof. 

b. Performance of these evaluation tasks under combat 
conditions (in the classroom or in the school) 
rather than in the atypical calm of a laboratory 
or research center. 

c. Provision of other alternatives for the innovative 
teacher or administrator than reading in the library 
about what others have done. 

d. Opportunity to learn evaluation by doing. 
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Part C —Formulate specific objectives . 

Example 6 — In-Sein;ice Training Directed to Problem Solving 
Through Clarification of Objectives 

(Within LEAs and SED) 

In various LEAs (through ESCOE and on-site visits) 
and in various units of SED (through interviews and 
conferences) BOER has developed the ability in 
participants in several areas: 

a. Clarification of goals and priorities by asking, 
•'What are you trying to do?*' and "Why?'' 

b. Stress upon the necessity to avoid vague and 
general terms which cannot be tied directly to 
concrete performance or observable activities. 

c. Communication pulling things together, e.g. , many 
problems lack precise objectives because those 
concerned never sat down together to look at the 
whole picture. 

d. Provision of a neutral perspective that demands 
that formulation of objectives be expressed in 
terminology easier to understand than the kind of 
jargon that sometimes hides the true meaning of a goal. 

Example 7 — Publications and Lectures on the Formulation of 

Clearly Stated and Easily Observable Behavioral Objectives 

(Within LEAs and SED, and available to the general public) 

Both by brief written handouts and by periodic availability 
of BOER staff for personal appearances, several practical 
advantages arise: 

a* Formulation of specific objectives for classroom 
Instruction by teachers under the watchful eye of 
BOER or ESCOE staff members. 

b. Editorial assistance and consultant services in the 
difficult area of expressing profundities in easily 
understood language. 

c. Awareness of the necessity of being able to communicate 
one's goals in a format and simplicity understandable 
by the general public. 

d. Development of the attitude that "vague goals are not 
very effective no matter how important they may be." 
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Part D Certify measures appropriate to objectives. 

Example 8 ^ Performance Tests 
(Within LEAs) 

BOER in cooperation with ESCOE and the R and D 
Centers has been able to develop: 

a* Criterion measures which measure a specific 
performance rather than norm measures which 
measure aptitude and ability. 

b. Methods to identify performance (which is subject 
to measurement and observation) rather than rely 
upon subject matter (which is not always visible 
in changed learner behavior) • 

c. Examinations which are chosen locally. This means 
that local autonomy in choice of objective is 
reinforced by local c?utonoruy in selection of avail- 
able test items correlate^ to these same objectives, 

d. Availability of evaluation instruments which offer 
performance options for testing rather than relying 
solely on the standard paper and pencil type of exam. 

Example 9 — Product Evaluation 

(Within LEAs and SED) 

BOER in cooperation with other SED units has been able 

to specify the results or product of a number of processes 

in such a way as to develop more objective measures, e.g.: 

a. Evaluation of a conference on the product (results 
and goals attained) rather than on the process 
(nice physical facilities and entertaining speakers). 

b. Evaluation of education progress for a particular 
student on the basis of product (v;hat he learned to 
do and achieve no matter how or v;here he learned it) 
rather than on the basis of process (discipline, 
attendance record, and source of learning). 

c. Distinction between essentials and inessentials in 
the learning process. Thus, teachers will approach 
evaluation on both the macroscopic and microFcopic 
approach but only when a particular level is appro- 
priate. 
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Fart E -^Assure utilization of evaluation products. 

Example 10 — Multi-Media Dissemination Beyond the Printed Page 
(Within LFAs and SED) 

In addition to making available printed and microfiche 
documents which sometimes go unread, BOER has been 
able to develop other ways to make educators aware 
of evalu.^cion research applicable to them; 

a. Phone calls on various ties to key educators 
calling their attention to recent breakthroughs 
ii^ evaluations similar to their own work. 

b. Periodic department conferences which stress BOER 
evaluation research v;hich can be replicated and 
used by other SED units. 

c. Personal appearances in selected schools through 
audience participation formats which encourage 
and permit on-the-spot application of highly 
specialized evaluation skills. 

d. Conferences which call together several hundred of 
the top specialists in a particular area (in 1971 
rehabila lators of ex-drug addicts and behavioral 
objective writers) to give facc-to-£ace examples 
of how to apply modern evaluative techniques in 
the effort to answer the question, "Are our efforts 
really achieving the results we think they are?" 

e. Films, recordings, slides, and video tape presen- 
tations have been used for greater visual impact. 

Example 11 — F ollowup Studies 

(Within LEAs) 

Rather than forget about projects designed to evaluate 
short-term goals, each funded agency or individual was 
asked several questions: 

^.t Did the research funded by us and performed by you 
have any long-term effects that are measurable? 

bt If yes, what are they and how do you measure them? 

c. If no, what went wrong once the funding stopped? 
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Part F —Maintain data storage, retrieval, and dissemination systems. 

Example 12 — Publications 

(Within LEAs and SED) 

In addition to the dissemination of documents provided 
by PREP, ERIC 3 and other microfiche services, BOER 
publishes documents on a number of themes: 

a. Current status of various components of the educa- 
tional see nej these can be used as benchmarks for 
future evaluation. 

b. Tools of research, current trends, and available 
resources. 

c. Characteristics studies of learners and school personnel. 

d. Media analysis of appropriate goals, and relevance studies. 

e. Studies of systems, subsystems, and programs. 

f. Periodic newsletters to give succinct overviews 
of current trends and major developments. 

Example 13 — Comput e rlz.ed R etriev al 

The time lag in a printed publication can often be fatal 
to the unanticipated needs of evaluation researchers. 
For this reason, it is often necessary to envision other 
and faster sources: 

a. Computerized literature searches* 

b. Computerized test writing according to guidelines 
and objectives chosen by an autonomous teacher, 

c. Instant availability of up-to-date and timely data 
possible only through a centralized computer easily 
accessible through local input-output terminals. 
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Section II - What BOER plans to continue to do for future progress in 
establishing an education evaluation system. 

Point of View of BOER 

It is self-evident that more time, money, and manpower would aid 
greatly in establishing an education evaluation system of the first 
caliber. However, this decision is not in the domain of the tasks and 
functions currently performed by BOER. 

Rather than develop into a lobby or a pressure group, BOER takes 
the position that being aware, of and striving for a unified education 
evaluation system is one of the best ways of eventually achieving this 
goal without unduly increasing dollar costs and without neglecting any 
of its current functions. 

Specific Definition of "being aware of" and "striving for " 

In addition to the evaluation component which is built into every 
piece of BOER-related research, BOER feels that it has developed insight, 
personnel, functions, skills, media, data, objective measures, appli- 
cations, systems, and other resources that can be integrated into almost 
any type of education evaluation system that is likely to emerge. 
These resources were not developed haphazardly; they grew out of concrete 
responses to documented educational needs and trends. In each specific 
instance, research and evaluation were developed which could go beyond 
merely local concerns and which would be implemented by other educators 
facing similar situations. 

All that is needed now is "linking pins" and "tooling up." 
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Specific Examples of *'linkinj^ pins " 

The basic approach to linking up these services need not 
necessarily start with an overall hierarchical structure superimposed 
on existing units. It might better start with coordination. 

Coordination basically means communication. Here are a few 
examples during the period, January to September 1971: 

a. BOER-RCU staff members, as a regular part of their 
functions, furnished consultation to more than 33 
units in the New York SED. 

b. Similarly, contacts were established with the New 
York Department of Labor and the Division of Employ- 
ment, Department of Civil Service, and other State 
agencies • 

c. Likewise, contacts were developed with USOE, National 
Institute of Mental Health, and other Federal agencies. 

In other words, coordination with other agencies does not require 

adding a person to the staff who does nothing else. It means that 

every staff member must be able to communicate what BOER is doing and 

that needless duplication be avoided by seeking out information and 

expertise available rather than trying to duplicate it needlessly on 

one's own. 

Pulling things together is not usually achieved by legislative fiat. 
A more efficacious method is in talking about one's job. BOER has 
numerous documented incidents in which needless duplication was avoided 
because a BOER staff member told other units and other government agencies 
what New York SED was doing. This resulted in two benefits: BOER gained 
from the experience of other units and agencies; the other units and 
agencies benefited from BOER's expertise; both units increased their 
effectiveness. In other words, the "multiplier effect" is a better 
•'linking pin" than the establishment of yet another super-bureaucracy or 
yet another specialist in communication. 




Specific Examples of "tooling up " 



Tooling up means a number of things. Here are general guidelines 
to be developed by specifics: 

a. The evaluation gap arises from tco much data in 
an undigested and unusable form. 

b. The evaluation system which will clo::e the evalu- 
ation gap is the system which will be understandable 
by the man-on-the-street . 

c. The man-on-the-street refers to people, e.g., teachers, 
administrators, graduate students, citizens, who are 
not professional evaluators. 

d. Attempts to disseminate this information must go 
beyond the distribution of more printed literature. 

a. The Evaluation Gap 

To bridge the evaluation gap, BOER has developed some general 

directives to writers of research evaluation: 

Be brief - The average consumer of research doesn't 
have enough time to spend more than 10 minutes with 
v;hat you put in front of his eyes. If your summary 
presentation points out clearly areas of practical - 
application, the reader v/ill then take the next 
step to seek out more detailed reports of evaluation 
systems and techniques. 

Be transmittable - If you have done a good job of 
v;riting up evaluation research, you will most likely 
be approached in person or by phone. Your compre- 
hension of the particular aspect in evaluation must 
be transmittable in a face-to-face exchange of ideas 
and questions. If your language is only compi'ehensive 
to specialists and professional evaluators, you will 
not be able to deliver the goods no matter how 
important and how practical your evaluation information 
may be. 

3« Be flexible - Replication means that someone will do 
your evaluation research over again in exactly the 
Same circumstances. Flexible adaptation means that your 
evaluation research will be applied in similar circum- 
stances and in answer to similar challenges. Once you 
use brief -to-the-point language instead of technical 
jargon, you will be ready to tailor and adjust evaluation 
to slightly different settings. 
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b. The Man-on-the-Street 



To reach the nonspecialist and the nonprofessional consumer of 
educational research, BOER has developed a few guidelines; 

1. Use simple language which can be understood by all. 

2. Try to use presentations which do not rely on a 
built-in necessity to use the lecture method, 

3. Stress the objectives and values of the particular 
aspect of evaluation being presented. 

c. The Nonprofessional Evaluator 

In presenting the nonspecialist with evaluation research information, 
several things must be remembered: 

1. Complicated presentations are not the best introduction. 

2. Educational leadership sometimes means spelling out 
how the consumer of research can apply it to his own 
situation. It is a mistake to presume that the non- 
specialist will see all the implications for himself. 

3. Since evaluation development really means staff 
development, it must be remembered that the first 
efforts of the nonprofessional evaluator will not 
result in perfectly finished products. It is often 
better to stress gradual improvement rather than 
unattainable ideals. 



d. Beyond Paper and Pencil and the Printed Page 

To stimulate educators to think of evaluation as something more active 
and penetrating than checklists and paper and pencil test questions, 
BOER has* developed other alternatives: 

1. Sometimes a phone call to the right person will do 
more co get someone to utilize an evaluation technique 
than will tons of documents received in the mail. 

2. Asking the question, "What one thing is most important 
among all the objectives you want to evaluate?" can 
often get the beginning evaluator to stress essentials 
rather than get bogged down in nonessentials. 



ERIC 



17 



CONCLUSION 



BOER-RCU staff members have gone through many evaluation research 

documents in the period June 1970 to October 1971. About 90 percent 

of these evaluations had the following conclusion: 

"All Project X needed for greater success was 
more money, time, personnel, and other resources," 

Such a conclusion is invariably true of any evaluation effort. 
A realistic approach would be to assume this and to try to do one's best 
with available resources. 

As pointed out above, the alternatives to lobbying and pressuring 
for more resources are not limited to doing nothing. 

Two concrete activities are spelled out above. One is to start 
coordinating existing services by exchange of information whenever possibl 
in an already busy schedule of activities. The other is to start tooling 
up existing functions and operations. One must realize that 

— evaluation starts with the beginning of any project 
and not at the end; 

— evaluation is best achieved when the goals of a 
particular function are expressed so clearly and 
succinctly that even a stranger walking in off the 
street can get a clear idea of what is going on and 
how well it is being done, 

•r in other words, evaluation is essentially linked 
to the development of clearljy stated , observable , 
and measurable .goal s , 

Most people know what they are trying to do whether or not they are 

evaluation specialists. The person who is becoming a better evaluator 

is able to clearly explain to a nonprofessional exactly what a particular 

function is all about. This is much better than substituting technical 

hardware for "brainware," 
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APPENDICES 
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1^ Massachusetts and New York Evaluation Service Center for Occupational 
Education, 

2. ERIC ... and the New York RCU 

3. Conference Feedback: Using Goal Awareness Instead of Sepaker-by-Speaker 
Evaluation 
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5. The Mini-Grant 
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9. Manpower Conference Feedback 
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to Teach Psychomotor Skills 

11a. Entry Requirements in Registered Apprentice Training 

lib. Career Programs in Two-Year Colleges 

11c. The Selection of Students for Entrance into Trade Programs in Public 
Secondary Schools 

lid. A Survey of Secondary School Graduates in New York State 

lie. A Study of Manpower Development and Training Act Programs in New York State 

llf. A Survey of Manpower Needs in the Commercial Nursey Industry 

llg. The Status and Role of Lunchroom Aides in Selected New York State School 
Districts 

llh. Final Report: Manpower Supply and Demand in Nassau-Suffolk 

Hi. Study of Articulation Program Between Nassau Community College and Topic 
House 

llj. Evaluation of Secondary School Programs to Prepare Students for Wage 
Earning in Occupations Related to Home Economics 
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^PPENDICEfc^ (cont.) 

Example 

12a* Measuring Relationships Among and Between Cognitive and Affective 
Behaviors in a Controlled Learning Situation 

12b. A Simulated Instructional Model for Educating Mentally Retarded Students f 
Employment in the Hotel-Motel Industry 

12c. An Investigation of the Reliability and Validity of Selected Occupational 
Competency Examinations and Their Use in Evaluating Prospective Trade and 
Industrial Teachers 

12d. A Followup of New York Sta High School Bookkeeping Students 

12e. A Comparative Study of High School Graduates from 1-Year, Full Day and 
2-Year, Half Day Vocational Programs in Suffolk County, New York 

12f. The Extent of Registered Apprentice Training in New York State 

13a. Automatic Business Data Processing in New York State Secondary Schools 

13b. Automatic Data Processing in New York State 

14a. Guidelines in Business Education 

14b. The Process and Product of Machine Shorthand Programs in New York State 
Schools 

14c. Elementary and Secondary Education Vouchers 

I4d. Cooperative Federalism 

14e. The Attitudes of School Board Members Toward Occupational Education 

I4f. Application of the Extended School Year Concept to a Board of Cooperative 
Educational Services Occupational Training Program 

I4g. The Impact of a Rescheduled School Year on Vocational Training Centers 
in New York State 

14h. Shared Educational Services in New York State Emphasis-Occupational 
Education Programs 
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ANNOTATED EXAMPLES 



Example #14a. 

This publication (Guidelines in Business Education) developes various 
guidelines and the rationale behind each .aspect of the evaluation of 
business education. 



Example #14b. 

This publication (The Process and Product of Machine Shorthand 
Programs in New York State Schools) developes guidelines for the 
inauguration of navj curricula. 

Example #14c. 

This publication (Elementary and Secondary Education Vouchers) stresses 
alternatives to educational financing in view of product evaluation by the 
ultimate consumers, parents, and students in school. 



Example #14d. 

This publication (Cooperative Federalism) points out the obvious link 
between Federal dollars and Federal evaluation. In this perspective 
evaluation is seen as a cooperative enterprise. 

Example 41^e, 



This publication (The Attitudes of School Board Members Toward 
Occupational Education) shows the link between school board attitudes 
evaluation and educational dollars. 

Example 4H4f. & g. 

These two publications (Application of the Extended School Year 
Concept wO a Board of Cooperative Educational Services Occupational 
Training Program & The Impact of a Rescheduled School Year on Vocational 
Training Centers in New York State) point out that evaluation must be 
something that begins with a project. If it is put off until the end, 
evaluation is done too late. 

Example #14h. 

This publication ( Shared Educational Services in New York State 
Emphasis--Occupational Education Programs) gives an example of criteria 
that have been developed on a cooperative basis. The point to note is 
the process of a criteria development. This same process will lead to 
many different criteria in different circumstances. 
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